Binary Neural Networks Algorithms, Architectures, and Applications (Baochang Zhang, Sheng Xu, Mingbao Lin etc.)

Principal Methods

FIGURE 1.3

A block in XNOR-Net.

XNOR-Net [199] changes the block structure in a typical CNN. A typical block in a

CNN contains diﬀerent layers: 1-Convolutional, 2-BatchNorm, 3-Activation, and 4-Pooling.

To further decrease information loss due to binarization, XNOR-Net normalizes the input

before binarization. This ensures the data have zero mean, so thresholding at zero minimizes

quantization error. The order of the layers in XNOR-Net is shown in Fig. 1.3.

The Bi-real Net [159] attributes the poor performance of 1-bit CNNs to their low rep-

resentation capacity. The representation capacity is deﬁned as the number of all possible

conﬁgurations of x, where x could be a scalar, vector, matrix, or tensor. Bi-real Net pro-

poses a simple shortcut to preserve real activations before the sign function to increase

the representation capability of the 1-bit CNN. As shown in Fig. 1.4, the block indicates

the structure “Sign →1-bit convolution →batch normalization →addition operator.” The

shortcut connects the input activations to the sign function in the current block to the

output activations after the batch normalization in the same block. These two activations

are added through an addition operator, and then the combined activations are passed to

the sign function in the next block.

The simple identity shortcut signiﬁcantly enhances the representation capability of each

block in the 1-bit CNN. The only additional cost of computation is the addition operation

of two real activations without additional memory cost.

BinaryDenseNet [12] designs a new BNN architecture that addresses the main drawbacks

of BNNs. DenseNets [92] apply shortcut connections so that new information gained in one

layer can be reused throughout the depth of the network. This is a signiﬁcant characteristic

that helps to maintain the information ﬂow. The bottleneck design in DenseNets signiﬁ-

cantly reduces the ﬁlters and values between layers, resulting in less information ﬂow in the

BNNs. These bottlenecks must be eliminated. Due to the limited representation capacity

of binary layers, the DenseNet architecture does not perform satisfactorily. This problem is

solved by increasing the growth rate or using a larger number of blocks. To keep the number

FIGURE 1.4

1-bit CNN with shortcut.